feat(desktop): add WebKit renderer recovery ladder for Linux - #3271
Open
wpfleger96 wants to merge 5 commits into
Open
feat(desktop): add WebKit renderer recovery ladder for Linux#3271wpfleger96 wants to merge 5 commits into
wpfleger96 wants to merge 5 commits into
Conversation
Some Linux GPU and compositor combinations crash the WebKit web process during startup, leaving Buzz with an invisible window and no way out (#2338, #2643). WebKit memoizes each renderer environment variable once per process, so a safer configuration can only ever be applied by launching a fresh process — the recovery is therefore built out of explicit relaunches rather than runtime toggles. On a crashed web process inside the startup window, the app prepares a durable episode record, releases the single-instance D-Bus name, proves the name is free, spawns a child with the next tier's exact environment, and exits. The child claims the record exclusively and confirms once it has started cleanly; a confirmed tier is reused on later launches. Recovery is crash-driven only. A window that never paints but also never crashes is not detected, because no available signal distinguishes it from a healthy window. Co-authored-by: Will Pfleger <[email protected]> Signed-off-by: Will Pfleger <[email protected]>
…lease boundary Two safety invariants in the recovery ladder were not enforced. A refusal raised after the single-instance name was released left the parent running without the name and unable to take it back, because the launcher's Result<(), Refusal> could not express which side of that irreversible boundary the refusal came from — and binary resolution, the one preflight most likely to fail, happened inside the launcher. Both refusal sides are now distinct outcomes, every preflight runs before the release, and a post-release refusal exits instead of lingering. The exclusive claim serialized claim-file creation, not record transitions: read-validate-write ran as three unsynchronized steps, so a delayed generation could stamp a stale receipt over a newer episode, a duplicated crash callback could spawn a second child, and a claim loser could reach the single-instance name ahead of the true owner. Every transition now runs under a crash-releasing advisory lock on a file outside the cleared directory, receipts must name the current (token, generation) and move the phase forward, the eligible-crash path is one-shot, and claim losers exit before Tauri is built. --safe-rendering against a user-set owned variable was silently dropped; it is now a fatal pre-Tauri error naming the conflicting assignments. Co-authored-by: Will Pfleger <[email protected]> Signed-off-by: Will Pfleger <[email protected]>
Boot decided from a record read outside the transaction lock and then mutated under it without re-checking that record was still current, so a decision could outlive the state it rested on: a concurrent --reset-rendering-mode could be undone by a stale retry recreating a prepared episode, and a delayed version-mismatch discard could delete a newly prepared current-version episode. Store::transition is now the only way anything durable changes. It re-reads the record under the caller's lock and writes only if the record still matches the state the caller decided from; a refusal makes boot re-decide from what is actually there rather than proceed. Because the expected source names a phase, the receipt chain is exact for free -- owned requires started, confirmed requires owned -- replacing a rank-based ordering that accepted skips. The prepare/release/spawn sequence moves to session::handoff, which keeps both files well clear of the size gate. Co-authored-by: Will Pfleger <[email protected]> Signed-off-by: Will Pfleger <[email protected]>
The two tests that inject a competing write from inside the spawn window shared one static path slot. The launch edge is a bare fn pointer and captures nothing, so the path has to reach it through a static -- but with one slot the tests overwrite each other's path whenever the harness runs them in parallel, which fails the assertions in CI while passing under a luckier local ordering. Verified by reintroducing the shared slot: 6/6 runs fail. With a slot per edge, 8/8 pass. Co-authored-by: Will Pfleger <[email protected]> Signed-off-by: Will Pfleger <[email protected]>
Compare-and-transition refuses a write whose source record moved, and the launch is then supposed to retake the decision against what is actually there. Nothing proved the retake happens: a launch that gave up on the refusal would have passed every existing test. The new case parks a launch on the state lock mid-handoff, replaces the record it classified with a confirmed tier 3, and asserts the handoff lands at tier 3 -- a tier only a second classification could have chosen. Verified by capping the reconcile loop at one attempt: this test is the only one that fails. Co-authored-by: Will Pfleger <[email protected]> Signed-off-by: Will Pfleger <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some Linux GPU and compositor combinations crash the WebKit web process during startup, leaving Buzz with an invisible window and no way out. WebKit memoizes each of its renderer environment variables exactly once per process (
AcceleratedBackingStore.cpp,std::call_once), so a safer configuration can only ever be applied by launching a fresh process — this recovery is built out of explicit relaunches, not runtime toggles.Closes #2338Scope
In: crash-driven recovery,
--safe-rendering,--reset-rendering-mode.Out: automatic no-paint detection. A window that never paints but also never crashes is deliberately not detected — no available signal distinguishes it from a healthy window, so that track stays gated on validation against affected hardware.
How it works
On a
Crashedweb-process termination inside the startup window, the app runs one explicit handoff:tauri::App::request_restartcannot serve here — it has no child-environment parameter and skips the single-instance name release, so every child born from it dies as a duplicate.std::env::set_varis equally out: unsound in a process with live GTK and Tokio threads, and WebKit would have memoized the variable already.Spawn success and a zero exit code are both worthless as survival signals, since a child that loses the name forwards its argv and exits 0. Only the child's claim receipt, correlated against the name's owner, proves a handoff happened.
Releasing the name is irreversible — the single-instance plugin cannot be re-registered — so the handoff outcome distinguishes a refusal raised before the release from one raised after. Every preflight that can fail, including resolving the binary to re-execute, runs before the release; a refusal there leaves the app running with its name. A refusal after it (the name turned out not to be free, or
spawnfailed) exits rather than lingering as a Buzz that no longer owns the name. The eligible-crash path is one-shot, and a recovery child that loses its claim exits before Tauri is built rather than competing for the name against the episode's true owner.Episode state
A durable record moves
prepared → started → owned → confirmed, with exclusive(token, generation)claims taken viacreate_new(O_EXCL). Claim files are never deleted — a stale claim is inert evidence that a generation already ran.An exclusive claim only serializes the creation of one claim file, which is not enough on its own: reading the record, deciding from it, and writing the next phase are three operations. So every transition runs under an advisory
flockon a lock file that is a sibling of the state directory (a lock inode inside it would be unlinked by--reset-rendering-modefrom under a concurrent holder).flockrather than anO_EXCLlockfile because the kernel releases it on process death — a lockfile left behind by a process that crashed mid-transition is exactly the failure this feature exists to handle.Serializing writers is still not sufficient, because a boot decision is taken from a read and acted on after a D-Bus round trip, so the record can move in between. Every durable change therefore goes through one compare-and-transition primitive that re-reads the record under the caller's lock and writes only if it still matches the state the caller decided from — otherwise the write is refused and boot re-decides from what is actually there. A concurrent
--reset-rendering-modecannot be undone by a retry decided before it, and a delayed version-mismatch discard cannot delete a newly prepared episode. A launch that is out-raced repeatedly stands down and runs its launched profile untracked, which writes nothing.Because the expected source names a phase, the receipt chain is exact rather than merely ordered:
ownedrequiresstartedandconfirmedrequiresowned, so aconfirmedthat skips a failedownedwrite is rejected instead of accepted as "forward". The same comparison rejects a receipt from a superseded episode, so aconfirmedracing a crash handoff cannot resurrect the episode the crash just disproved.exhaustedis a separately validated terminal write against the whole attempt, and nothing leads out of it. Ownership is correlated against the bus (GetConnectionUnixProcessID, unique name,GetId) asabsent | correlated | mismatched | uncorrelatable; an owner that exists but cannot be tied to the record is a handoff failure, never a free name.ownedreceipts are bound to the bus id, because a unique name like:1.4only identifies a connection on the bus that issued it.Tier table
full-gpushm-transportWEBKIT_DMABUF_RENDERER_FORCE_SHM=1cpu-raster+ WEBKIT_SKIA_ENABLE_CPU_RENDERING=1no-accelWEBKIT_DISABLE_DMABUF_RENDERER=1,WEBKIT_SKIA_ENABLE_CPU_RENDERING=1x11-fallback+ GDK_BACKEND=x11WEBKIT_DISABLE_COMPOSITING_MODEis owned (a user value opts out) but no tier sets it, pending the 3-var vs 4-var isolation on #2643.One crash advances exactly one rung and no rung is ever repeated, so the tier count is the per-package attempt cap — no separate counter can drift from it.
Environment ownership
Package-scoped. AppImage owns the 4
WEBKIT_*variables; native additionally ownsGDK_BACKEND, which on AppImage is a package invariant re-exported by linuxdeploy's GTK AppRun hook before the binary starts and therefore never a user signal.Any genuine user assignment of an owned variable — any value, including
0and empty — disables recovery wholesale with a log line.--safe-renderingcombined with such an assignment is refused as a fatal pre-Tauri error naming the conflicting variables, rather than guessed either way: honouring the flag would overwrite configuration the user typed, and honouring the environment would silently ignore a rescue flag from a user whose app will not start. Recovery children are taggedBUZZ_RENDER_EPISODE/BUZZ_RENDER_GENERATION/BUZZ_RENDER_PROFILEso they never reinterpret an injected profile as user configuration.AppImage verification
Built
Buzz_0.4.26_aarch64.AppImageand ran it underxvfb-run dbus-run-session. The crash edge is driven bySIGKILLon the WebKit web process inside the eligibility window — the sameCrashedtermination the reporters hit.Crash-driven walk from a clean state — all four AppImage rungs, then terminal:
Exactly one survivor, and it is the D-Bus name owner:
Its environment is the tier and nothing inherited:
The record ends
exhaustedat tier 3 naming that pid, with one claim file per generation and the lock file a sibling of the state directory:Flags on the same build:
--safe-renderingforcesno-acceland leaves no state directory behind;WEBKIT_DISABLE_DMABUF_RENDERER=0in the user's environment disables recovery wholesale and persists nothing;--safe-renderingunder that same assignment exits 1 before Tauri with the conflict named and nothing persisted;--reset-rendering-modeclears the record and exits without launching, and reports accurately on a second run.Layout
desktop/src-tauri/src/render_recovery/—profiles(package scoping + tier table),state(durable record, exclusive claims, the compare-and-transition primitive),episode(attempt identity + pure ladder rules),dbus(name observation),classify(record + observation → decision),launcher(the explicit relaunch),cli(flags),session(live process, withsession::handofffor prepare/release/spawn),wiring(Tauri plugin). 65 unit tests cover the three generation races, injected deaths at all four points, the per-package cap, the version-scoped crash-free-startup fact, and the exact receipt chain including every phase skip.The races that need real contention are driven rather than simulated. Two same-generation children race as two real processes through a filesystem barrier, so their pids differ and the durable receipt can be checked against the identity of the one that won. A stale generation claiming while a newer one is prepared, a confirmation against a crash handoff, two termination callbacks, and a rollback against a later episode are driven from two threads through a barrier with the state lock held across the window each one needs, and each asserts the second writer actually blocks. A reset racing a decision taken before it, and a delayed version-mismatch discard against a fresh episode, assert the write is refused and the record is left intact.
lib.rsgains 37 lines: the boot call before Tauri is built, the four boot outcomes it must act on, and plugin registration after the single-instance plugin.Related:
desktop/scripts/fix-appimage.shcarries no rendering environment exports and is unchanged here — the ladder owns those variables exclusively.